Add test for selecting a single complex field array and its parent struct array [databricks] #8744

razajafri · 2023-07-18T19:03:34Z

PR tries to mimic the test in Spark's SchemaPruningSuite.

We create a table with complex fields and try to read only a subfield to see if column pruning is working the way it's supposed to i.e. we are not reading unnecessary columns. e.g. if we have a complex type contact with an array of friends, like so

Contact {
   first_name: String
   middle_name: String
   last_name: String
   friends: Array[Contact]
}

selecting spark.table("contacts").select(explode("friends").alias("friend").select("friend.first_name") shouldn't also read the middle_name and last_name of the friends field.

fixes #8712
fixes #8713
fixes #8714
fixes #8715

Signed-off-by: Raza Jafri <[email protected]>

mythrocks · 2023-07-18T19:59:58Z

It would be good for description in this PR (if not also the issue #8712) to be more descriptive of the change.

The test here is a sampling of tests covered in SchemaPruningSuite. The objective is to cover some scenarios where column pruning might be tripped up. SchemaPruningSuite does this by selecting different combinations of column (and sub-column) projections.

You should consider combining these tests into a single change. All that's changing between these tests is the output projection.

mythrocks · 2023-07-18T20:47:16Z

Adding to this for clarity:

There are 4 scenarios listed in [TEST] Compatibility tests for data formats #8666, sampled from SchemaPruningSuite. I don't think it's necessary to exhaustively run all the cases in SchemaPruningSuite. We just need a few (like these 4) to test that the pruning is working.
Some effort will be needed to do the equivalent of checkScan and checkScanSchemata. We'll just need to check that the file-scan operator in question has the expected read schema, just like in the test.

…a-pruning-test

razajafri · 2023-07-22T00:52:23Z

build

razajafri · 2023-07-25T01:23:44Z

build

razajafri · 2023-07-25T17:47:36Z

connection failure in premerge

integration_tests/src/main/python/prune_partition_column_test.py

sql-plugin/src/main/scala/org/apache/spark/sql/rapids/ExecutionPlanCaptureCallback.scala

integration_tests/src/main/python/prune_partition_column_test.py

razajafri · 2023-07-26T19:51:47Z

build

razajafri · 2023-07-27T00:14:41Z

premerge failing with the following Exception which seems to be unrelated to this change. I also see another PR failing with the same exception

[2023-07-26T21:17:28.616Z] [1m[31m../../src/main/python/parquet_testing_test.py[0m:116: in <module>
[2023-07-26T21:17:28.616Z]     @pytest.mark.parametrize("path", gen_testing_params_for_valid_files())
[2023-07-26T21:17:28.616Z] [1m[31m../../src/main/python/parquet_testing_test.py[0m:105: in gen_testing_params_for_valid_files
[2023-07-26T21:17:28.616Z]     for f in locate_parquet_testing_files():
[2023-07-26T21:17:28.616Z] [1m[31m../../src/main/python/parquet_testing_test.py[0m:91: in locate_parquet_testing_files
[2023-07-26T21:17:28.616Z]     raise AssertionError("Cannot find parquet-testing data in any of: " + locations)
[2023-07-26T21:17:28.616Z] [1m[31mE   AssertionError: Cannot find parquet-testing data in any of: /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-7623-ci-2/integration_tests/src/test/resources/parquet-testing/data/*.parquet, /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-7623-ci-2/integration_tests/src/test/resources/parquet-testing/bad_data/*.parquet, /home/jenkins/agent/workspace/jenkins-rapids_premerge-github-7623-ci-2/thirdparty/parquet-testing/data/*.parquet, /home/jenkins/agent/workspace/jenkins-

razajafri · 2023-07-27T00:14:46Z

build

pxLi · 2023-07-27T01:12:36Z

build

revans2 · 2023-07-27T18:22:28Z

@razajafri try to upmerge and then run the premerge tests again.

…a-pruning-test

razajafri · 2023-07-27T19:45:05Z

build

…a-pruning-test

razajafri · 2023-07-31T20:01:30Z

I am looking into the CI failure. From the logs, the test that I added is possibly crashing the JVM which is causing
subsequent test failures but everything passes on my local workstation.

It could just be a coincidence that the connection dropped so I will try to kick this off one more time.

razajafri · 2023-07-31T20:01:34Z

build

razajafri · 2023-07-31T21:51:12Z

Self-bootstrap failure during launch. Please try again later and contact Databricks if the problem persists. Node daemon fast failed and did not answer ping

razajafri · 2023-07-31T21:51:16Z

build

pxLi · 2023-08-02T00:34:26Z

build

pxLi · 2023-08-02T03:06:30Z

build

…a-pruning-test

razajafri · 2023-08-04T21:09:09Z

build

razajafri · 2023-08-07T22:52:46Z

build

razajafri · 2023-08-08T05:36:35Z

@mythrocks @revans2 can you please take another look

…arent struct array [databricks] (NVIDIA#8744)" This reverts commit 1fb5fc4. Signed-off-by: Jason Lowe <[email protected]>

* Revert "Create an anonymous subclass of AdaptiveSparkPlanHelper in ExecutionPlanCaptureCallback.scala [databricks] (#8977)" This reverts commit bcc929c. Signed-off-by: Jason Lowe <[email protected]> * Revert "Add test for selecting a single complex field array and its parent struct array [databricks] (#8744)" This reverts commit 1fb5fc4. Signed-off-by: Jason Lowe <[email protected]> --------- Signed-off-by: Jason Lowe <[email protected]>

razajafri added 3 commits July 17, 2023 17:32

added a test for schema pruning for orc

d896b56

added a test for schema pruning for parquet

939bbc9

signing off

faf6a52

Signed-off-by: Raza Jafri <[email protected]>

razajafri self-assigned this Jul 18, 2023

razajafri marked this pull request as draft July 18, 2023 20:03

razajafri added 2 commits July 21, 2023 17:38

Merge remote-tracking branch 'origin/branch-23.08' into SP-8712-schem…

4a441af

…a-pruning-test

Add schema pruning tests

8bd9523

razajafri marked this pull request as ready for review July 22, 2023 00:52

add test for missing complex field

302f20f

revans2 reviewed Jul 25, 2023

View reviewed changes

razajafri added 2 commits July 25, 2023 18:48

addressed review comments

5e31166

added comments

ec75ae6

pxLi mentioned this pull request Jul 26, 2023

Bump up dep versions to 23.10.0-SNAPSHOT [databricks] #8805

Merged

revans2 previously approved these changes Jul 26, 2023

View reviewed changes

integration_tests/src/main/python/prune_partition_column_test.py Show resolved Hide resolved

integration_tests/src/main/python/prune_partition_column_test.py Show resolved Hide resolved

sameerz added the test Only impacts tests label Jul 26, 2023

Merge remote-tracking branch 'origin/branch-23.08' into SP-8712-schem…

e4079c0

…a-pruning-test

Merge remote-tracking branch 'origin/branch-23.08' into SP-8712-schem…

5b72769

…a-pruning-test

razajafri added 2 commits August 2, 2023 12:12

Merge remote-tracking branch 'origin/branch-23.08' into SP-8712-schem…

26059dd

…a-pruning-test

skip failing method instead of xfail

ed84eb2

razajafri dismissed revans2’s stale review via ed84eb2 August 4, 2023 20:57

use the new tmp table name

887272e

revans2 approved these changes Aug 8, 2023

View reviewed changes

razajafri merged commit 1fb5fc4 into NVIDIA:branch-23.08 Aug 8, 2023

razajafri deleted the SP-8712-schema-pruning-test branch August 8, 2023 16:49

jlowe mentioned this pull request Aug 11, 2023

Revert test changes to fix binary dedup error [databricks] #9006

Merged

razajafri mentioned this pull request Aug 11, 2023

Add test for selecting a single complex field array and its parent struct array [databricks] #9018

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add test for selecting a single complex field array and its parent struct array [databricks] #8744

Add test for selecting a single complex field array and its parent struct array [databricks] #8744

razajafri commented Jul 18, 2023 •

edited

Loading

mythrocks commented Jul 18, 2023

mythrocks commented Jul 18, 2023 •

edited

Loading

razajafri commented Jul 22, 2023

razajafri commented Jul 25, 2023

razajafri commented Jul 25, 2023

razajafri commented Jul 26, 2023

razajafri commented Jul 27, 2023

razajafri commented Jul 27, 2023

pxLi commented Jul 27, 2023

revans2 commented Jul 27, 2023

razajafri commented Jul 27, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

pxLi commented Aug 2, 2023

pxLi commented Aug 2, 2023

razajafri commented Aug 4, 2023

razajafri commented Aug 7, 2023

razajafri commented Aug 8, 2023

Add test for selecting a single complex field array and its parent struct array [databricks] #8744

Add test for selecting a single complex field array and its parent struct array [databricks] #8744

Conversation

razajafri commented Jul 18, 2023 • edited Loading

mythrocks commented Jul 18, 2023

mythrocks commented Jul 18, 2023 • edited Loading

razajafri commented Jul 22, 2023

razajafri commented Jul 25, 2023

razajafri commented Jul 25, 2023

razajafri commented Jul 26, 2023

razajafri commented Jul 27, 2023

razajafri commented Jul 27, 2023

pxLi commented Jul 27, 2023

revans2 commented Jul 27, 2023

razajafri commented Jul 27, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

razajafri commented Jul 31, 2023

pxLi commented Aug 2, 2023

pxLi commented Aug 2, 2023

razajafri commented Aug 4, 2023

razajafri commented Aug 7, 2023

razajafri commented Aug 8, 2023

razajafri commented Jul 18, 2023 •

edited

Loading

mythrocks commented Jul 18, 2023 •

edited

Loading